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Summary of Review 

The report, Yearning to Break Free: Ohio Superintendents Speak Out, describes findings of a 
survey of 246 Ohio school superintendents about critical issues facing the state’s educational 
system. In particular, the intent of the study was to examine how superintendents might do 
more with fewer resources. The authors conclude that Ohio districts need increased managerial 
authority and control over state revenue streams and other funds; transformed collective 
bargaining rights; and repeal of automatic increases in teacher salaries. Such efforts, they argue, 
are imperative to improve the educational opportunities of all children in the state. But the 
authors do not provide evidence to support this latter claim. The combined effects of non- 
representative sampling, loaded or inappropriately worded items, and the conflating of opinion 
and fact make the report’s conclusions problematic. Myriad factors contribute to student 
achievement, including home and community effects, campus resources (material and non- 
material), as well as teacher competence which are not examined or considered. And, despite the 
reported finding that superintendents prefer greater autonomy in personnel and school policies 
over increased funding, the majority of superintendents also contend that they would see a 
trade-off of more autonomy with a decrease in funding as undesirable. While the report’s main 
thrust is to justify flat or reduced spending, the report lacks sufficient rigor to make it useful to 
guide policy or practice. 
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Review of Yearning to Break Free: 
Ohio Superintendents Speak Out 

Catherine Horn and Gary Dworkin, University of Houston 



I. Introduction 

The report, Yearning to Break Free: Ohio Superintendents Speak Out, co-written by the 
Thomas B, Fordham Institute and the FDR Group, ^ describes findings of a survey of 246 Ohio 
public and regional Education Service Center Superintendents as well as charter school leaders 
on critical issues facing the state’s educational system. The authors conclude that Ohio districts 
need increased managerial authority and control over state revenue streams and other funds; 
transformation of collective bargaining rights; and repeal of automatic increases in teacher 
salaries. Such efforts, they argue, are imperative in improving the educational opportunities of 
all children in the state. 



II. Findings and Conclusions of the Report 

The report draws several interrelated conclusions from the survey responses. First, the authors 
note that a “lack of money is not the bigger problem facing education...; it’s how and on what 
the money is spent” (p. 10). They further conclude that increasing administrative autonomy over 
staffing decisions would lead to increased academic gains for students. Third, the study suggests 



Merely because the superintendents think that getting greater control over 
the hiring and firing of teachers will raise tests scores is not the same as 
empirical evidence that such a practice will raise scores. 



that, in general, the collective bargaining system in Ohio needs state-mandated revisions. 
Specific changes are also needed in state laws governing staffing and salary. 

As cost-saving strategies, the authors of the study recommend that “two promising ways to save 
districts money are to give superintendents greater control over combined state revenue streams 
and to mandate a statewide health insurance plan for the K-12 system” (p. 11). The study also 
reports that school superintendents share many of the same attitudes as Ohio’s Educational 
Service Centers (ESC) superintendents and leaders of charter schools. They report, though, that 
ESC superintendents were more likely than other superintendents to see “the misuse of 
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resources rather than the lack of funding” (p. ii) as the central problem in education. Charter 
school leaders see the central problem as “how and where the money is spent” (p. 12). 



III. The Report’s Rationale for Its Findings and Conclusions 

In the forward to the report, Chester Finn states that “Ohio simply can’t afford not to seek 
dramatic achievement gains and gap reductions, no matter how tough the fiscal situation is” (p. 
5), but in the face of constricting budgets, 

This can only happen if those leaders have the capacity and the authority to act on their best 
judgment of what their teachers and students need. If the state shackles them with rules and 
envelops them in mandates even as it cuts their budgets, achievement will inevitably head 
down, not up. The same is true of teacher (and other employee) contracts that force them to 
spend scarce money in educationally unproductive ways because of provisions attuned to the 
interests of adults rather than students (p. 5). 

Finn says the survey was administered in order to give a public voice to privately articulated 
concerns “without making trouble for individual superintendents” (p. 6). To achieve that end, the 
report utilizes simple descriptive item-by-item reporting procedures. No inferential statistics 
were included nor was any additional supporting data from other sources used to support the 
recommendations . 



IV. The Report’s Use of Research Literature 

The report makes no use of available peer-reviewed literature to either substantiate the 
development of the survey instrument or to support the validity of the conclusions. The 
omission of supporting literature is particularly surprising given the broad body of available 
empirical work that has studied the full range of substantive issues addressed in this report. ^ 
Policy makers and other readers, then, are left to guess about (or assume without challenge) the 
appropriateness, utility, or generalizability of the conclusions drawn from this study. 

V. Review of the Report’s Methods 

This study suffers from several serious design flaws. Each is discussed in turn, but in sum, these 
various miscalculations fundamentally weaken the reader’s ability to draw useful conclusions. 

Response Rates and Sample Bias 

First, with only 41% of invited participants responding, the ability to generalize from the 
findings is weak. The amount and type of response bias based on differences between those who 
did and did not respond is simply unknown. 

Additionally, the study primarily presents aggregated responses for superintendents and charter 
leaders. Yet there is reason to expect that the charter leaders may have opinions distinctly 
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different from local public school districts. When the study does present disaggregated 
information, it relies on a charter school sample that had only a 15% response rate. Both return 
rates make it impossible to accurately estimate the range of opinions of administrators in these 
different settings. Nevertheless, the authors draw conclusions from both the total and 
subsamples as if they were adequate representations of those groups. 

The study also fails to provide a consistent comparison between sample and population 
characteristics, leaving the reader unable to determine whether sampling biases distort the data. 
For example, the report notes population statistics with respect to location (i.e., urban, 
suburban, small town, and rural), student enrollment, and Ohio Department of Education 
(ODE) ratings (e.g.. Excellent, Effective, Continuous Improvement), but fails to do the same 



The study also fails to provide a consistent comparison between sample 
and population characteristics, leaving the reader unable to determine 
whether sampling biases distort the data. 



with respect to the percentage of students within a district who are economically disadvantaged. 
The reader is left to wonder why some but not all of the readily available ODE contextual 
information is provided. No information is provided about non-respondents and how they may 
have differed demographically from the obtained sample. 

Bridge contends that sound sampling practice requires consideration of at least nine 
structural/demographic variables to discount the likelihood of sampling bias . 3 In the Ohio case, 
the ODE in fact uses a complex multi-categorical classification of its districts. “The purpose of 
developing a typology of districts is to provide a rational basis for making data-driven 
comparisons of groups of districts. Such groups include districts that share certain demographic 
characteristics. As a result, the groups can serve as a basis for a stratified sample of districts in 
the state . ”4 The authors emphasize the multiple forums they organized throughout the state in 
order to ensure that the survey was grounded within the Ohio context. However, their design 
failed to incorporate the essential organizational framework commonly employed in the state. 
Such omissions draw into question the depth with which the Ohio context was thoroughly 
considered. 

Instrument Design 

The study uses a 45-item questionnaire primarily comprised of closed-response items 
supplemented by two open-response questions. The instrument asks questions intended to 
measure superintendents’ perceptions on “the most critical issues facing K-12 education in the 
Buckeye State” (p. 1), including the relative cost savings of various changes to Ohio’s laws; the 
likelihood that certain choices would bring about substantial cost savings; and obstacles to 
improving public education. The selection of items as indicators of these “critical issues” 
suggests a distinct political agenda. The reader is left with the impression that narrowly selected 
items with biased wording were designed to justify an attack on collective bargaining statutes. 
For example: 
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• Some questions force respondents to choose between inappropriate dichotomies, such 
as: 

“If you had to choose between these two things, which do you think would be MORE 
likely to lead to improvement in student achievement in your district? Significant 
increases in school funding / Significant expansion of management” (p. 28). 

• Other survey items include language that is leading. Take, for example, the following 
question: 

“I may dislike mandates, but some of the problems facing Ohio’s school districts require 
state legislation” (p. 31). 

• Additionally, some questions use vague or nebulous language. 

“State law that [sic] permits district-labor negotiations over a variety of workforce 
issues that really should be off the table” (p. 30). 



Commonly accepted item development practice admonishes against all of these approaches. 
Reliable and valid information is produced only by surveys including clearly worded items that 
provide an opportunity for a full range of responses. s In contrast, the survey includes items that 
severely limit a respondent’s ability to express a full range. For example, items that ask the 
superintendents to rank order the effects of legislative changes that weaken teachers’ rights do 
not also include perceived negative effects for schools. 

The study reports that the instrument has a substantial seven-point overall margin of error and 
even larger margins for subgroups, but provides no psychometric properties for any of the items. 
No detail is included about how the standard error was computed, what those “larger margins” 
among subgroups were, or whether the error distinctions differed item-by-item or were based 
on aggregate scores. Overall, the serious limitations of the survey fundamentally challenge 
readers’ ability to draw serious conclusions about the perspectives of Ohio public 
superintendents, ESC superintendents and charter school leaders. 

Confounded Sample Selection and Analysis 

Concentrating only on superintendents to the exclusion of other groups also introduces a 
selection bias in that superintendents, by structural design, will have different views of collective 
bargaining than will principals, school boards or teachers. 

Further, the design of the study design does not allow for any understanding of variations within 
districts, even among charter schools and the regional ESCs. Rather, the study presents district 
perspectives as if they were entirely homogeneous. Understanding between district differences 
is most effectively done by also accounting for differences within districts.^ Even a cursory 
analysis of accountability data from Ohio indicates that substantial discrepancies often exist 
within districts around key student performance indicators. 7 
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Improperly Aggregating Disparate Groups 



The study’s design also makes a poor decision to combine local public school district 
superintendents with charter school leaders and ESC regional superintendents in representing 
the findings. Charter school leaders, for example, were more likely than others to think that 
student achievement can be improved with “significant increases in school funding,” rather than 
“significant expansion of management authority over staff’ (p.12). However, as charter schools 
are granted more autonomy in diverse areas, including staffing rules, it would be expected that 
their responses would differ from traditional public school superintendents. Because the vast 
majority of findings merge these groups, generalizing from the data is not warranted. 

Fundamentally, the study’s findings are compromised by the failure to incorporate relevant and 
extensive educational research, including a plethora of studies on sampling theory^ and survey 
design .9 



VI. Review of the Validity of the Findings and Conclusions 

In general, the authors conclude that superintendents believe greater autonomy, fewer 
mandates, and an opportunity to hire and fire at will will result in improved achievement test 
scores, but they do not provide evidence to support this claim. The improved achievement 
claims are not addressed at all. Further, this imperfectly designed survey is neither sufficient 
nor compelling. Myriad other factors that make strong contributions to student achievement. 



The combined effects of non-representative sampling, loaded or 
inappropriately worded items, and the conflating of opinion and fact, 
however, make the report’s conclusions problematic. 



including home and community effects, campus resources (material and non-material), and 
teacher competence, are not examined or considered. Examining narrow survey data tells the 
reader nothing about the importance or the effects of the recommendations in a school context. 
Additionally, the study treats opinions as statements of facts. Merely because the 
superintendents think that getting greater control over the hiring and firing of teachers will raise 
tests scores is not the same as empirical evidence that such a practice will raise scores. 

That said, some aspects of the report are not surprising. School superintendents are charged by 
their school boards and the public with improving student achievement in a high-stakes 
environment while keeping school taxes low. Consequently, unfunded mandates and employee 
contract negotiations represent threats to the balance of the competing pressures on 
superintendents. Thus, it is entirely predictable that superintendents would like to have greater 
control over both teachers’ salaries and state regulations. Further, while many of the 
superintendents reported that they were unhappy with their school boards for refusing to 
support “get tough” policies in dealing with teachers and teacher unions, it is unlikely that the 
state or certainly the school boards will grant greater autonomy to superintendents with respect 
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to board prerogatives. Furthermore, it presumes that school superintendents would universally 
agree that “get tough” policies are sound educational practice. 

In this survey, when phrased in a general fashion, the superintendents expressed a greater 
support for forgoing funding in exchange for increased autonomy in decision making. However, 
when the questions turned more specifically to their own districts, they saw the lack of funding 
or decreased funding as disastrous. Despite the reported finding that superintendents prefer 
greater autonomy in personnel and school policies over increased funding, the majority of 
superintendents also contend that they would see a trade-off of more autonomy with a decrease 
in funding as undesirable. 



VII. Usefulness of the Report for Guidance of Policy and Practice 

The intent of the study was to examine how superintendents might do more with less. The study 
concludes that superintendents would be able to raise student achievement, hence district 
accountability profiles, if they had more autonomy in their dealings with teachers and teacher 
unions and were freed from some state mandates. The combined effects of non-representative 
sampling, loaded or inappropriately worded items, and the conflating of opinion and fact, 
however, make the report’s conclusions problematic. This is a survey done at a moment in 
history in which both state legislatures and the federal government have reduced school 
funding. Reality suggests more funding is unrealistic. The choice between miniscule increases in 
budget or more control over spending might lead superintendents to choose the latter at this 
time and within this set of circumstances. But that condition does not address or answer the 
question of whether such a decision will raise student achievement. Nor does it tackle the 
question of what fiscal and programmatic post-recession choices superintendents might make if 
those increases are linked to some loss of autonomy. In total, the report lacks sufficient rigor to 
make it useful to guide policy or practice. 
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